The Risks of Generative AI Agents to Financial Services

Introduction

One day soon, you may be able to ask Siri to pay your cell phone bill from funds in your checking account. Or ask Alexa to recommend investments tailored to your risk profile. Or tell Gemini to manage your investing portfolio so you can travel in retirement. Such capabilities will be possible thanks to generative artificial intelligence, or Generative AI. Generative AI agents are computer systems with abilities to interpret and execute requests, such as these examples, without additional human interaction, and have been described as “the next frontier of Generative AI” (Yee et al. 2024). They have the potential to change the way individuals and firms interact with their banks and other financial services providers, opening the door to efficiency and economic growth—but also posing new risks to consumers, investors, and the safety and soundness of the financial system.

Generative AI agents may assist retail consumers with wealth management, such as working through personal assistants like Siri and Alexa to recommend financial products or serving as robo-advisors tailored to investors’ needs. They may also help nonfinancial firms through off-the-shelf treasury management solutions that balance financial returns with liquidity needs, and help financial institutions with risk management and regulatory compliance, such as with automated fraud detection, customer identity verification, and risk assessments through highly customized software (Zheng et al. 2019; Polak et al. 2020; OECD 2021).

Simultaneously, Generative AI agents threaten to destabilize the financial system, sending it swinging from crisis to crisis. Malicious actors can use it to defraud consumers, execute cyberattacks against financial institutions, and engage in market manipulation (Fang et al. 2024b; Mizuta 2020; Hsu 2024). Financial institutions’ own uses of Generative AI can hallucinate (that is, produce false or misleading outputs), resulting in harms to their customers, the institutions themselves, and the financial markets in which they operate (CFTC 2024). And when individuals and real-economy firms rely on a small number of Generative AI providers for financial decisions, they can engage in herding behavior that results in bank runs or flash crashes (Gensler and Bailey 2020).

Generative AI, in its current form, is not “good” or “bad” in and of itself. The large language models that power AI agents are simply computer systems capable of generating new content, such as images, text, audio, or video from a simple prompt. These systems are best considered a form of applied statistics—they capture patterns in the data upon which they have been trained and create outputs that resemble the training data but are unique variations. These software are just the latest use of algorithms and machine learning techniques that have been used in financial markets for decades, unlike in other industries in which Generative AI is novel, and can be used as inputs in human decision-making, as copilots that make decisions in coordination with humans, and as agents that make decisions on behalf of humans (US Senate Committee on Homeland Security and Governmental Affairs 2024; Hsu 2024).

To that end, the concern in this brief is not simply about the use of algorithms in finance, but about a world in which AI agents are widely available to individuals and small businesses as well as the largest financial firms; in which malicious actors may easily use Generative AI to scam financial institutions and their customers; and in which financial institutions use Generative AI to interact with their customers, rather than human employees.

In particular, this brief is concerned about the harms that may result from individuals’ and small businesses’ AI agents interacting with large financial institutions and scammers—especially if they rely on developers’ statements that AI agents will act in customers’ interests.

Some United States financial regulators are already working to address the harms Generative AI poses. The Consumer Financial Protection Bureau (CFPB) has explained that federal law does “not permit creditors to use complex algorithms when doing so means they cannot provide the specific and accurate reasons for adverse actions” and has penalized financial institutions for relying on faulty automated compliance systems (CFPB 2022a; CFPB 2022b). The federal banking agencies have created offices to study financial innovation and AI (Phillips and Conner 2024). The Securities and Exchange Commission (SEC) has proposed a regulation addressing brokers’ uses of Generative AI and has begun examining investment advisers’ uses of Generative AI for offering financial advice (US Securities and Exchange Commission 2023).

Nevertheless, more needs to be done, especially as it comes to the use of AI agents in the financial system. This brief highlights the expected rise of AI agents and the risks that their use by financial institutions, real-economy firms, and individuals pose to all aspects of the financial system and the families and businesses who rely on it. It concludes with recommendations to Congress and regulators.

Suggested Citation

Related Resources